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High Fidelity: Investing in Evaluation Training 


How are states providing training for new teacher and principal 
evaluation systems? 

High-quality training is a crucial investment in establishing and maintaining implementation fidelity 
as well as building educators’ trust in the new process. Training approaches for educator evaluation 
vary both in format (i.e., how it’s delivered) and content (i.e., what is provided). Train-the-trainer 
sessions, online professional learning modules, videos, webinars, in-person presentations — these 
are all examples of the formats that states use to provide training on new performance evaluation 
systems. In this edition of Ask the Team, we report on both the format and the content of state 
approaches to evaluation training. Specifically, our scan of state approaches to training turned up 
three categories of content: 

■ District leadership training empowers and supports district leadership teams by providing 
an overview of state-level legislative and policy changes and providing information, tools, 
and resources for implementation planning. 

■ Educator orientation training informs and prepares school-based educators who are being 
evaluated by providing an overview of the new system and detailed information, materials, 
and tools for completing the new evaluation process. 

■ Evaluator training cultivates and assesses the skills and knowledge of teacher and principal 
evaluators through an in-depth dive into instructional and leadership rubrics and practicing 
data collection and scoring. 

States and districts are testing a wide variety of format and content combinations for providing 
training on evaluation, but little is currently known about their efficacy, particularly at scale. For now, 
we offer the following state examples as a starting point for your state’s deliberations regarding 
evaluation training. 
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DISTRICT LEADERSHIP TRAINING 


District leadership teams need knowledge, tools, and 
strategies for implementing new educator evaluation 
systems with fidelity. At district leadership trainings, 
participants typically learn about the broad features 
of the new evaluation system; new legal and policy 
requirements; implementation timelines; and key tools, 
materials, processes, and procedures (or information 
and resources for designing them) for a district. 

States use a variety of methods, often in combination, 
to provide this training, such as the following: 

■ Statewide conferences. New York held a two-day conference for regional and district 
leaders that provided turnkey training to support districts in understanding the state’s 
new evaluation requirements, developing evaluation system plans, and accessing state 
and regional resources for completing the process. 

■ Online video or webinar resources. Indiana created DOE Dialogue: Staff Performance 
Evaluation, an 11-minute video that summarizes the new legal requirements for performance 
evaluation, explains the state’s process for reviewing and approving district evaluation plans, 
and gives district leaders a checklist for the review process. 


State 

^Spotlight 


■ The symposium provided information on the state’s model evaluation system as well as alternative models, 
resources for identifying evaluation measures, and presentations by principals and district leaders already 
implementing new evaluation systems in the state. 

■ After the event, Ohio created a short video of highlights from the symposium and integrated these video clips into 
two online professional learning modules that introduce the Ohio Teacher Evaluation System. The modules include 
a narrated PowerPoint presentation and short video clips of Ohio principals describing how they have implemented 
specific aspects of the model in their districts or schools. 

Source: Teacher Evaluation System in Ohio (http://www.ode.state.oh.us/GD/Templates/Pages/ODE/ODEDetail. 

aspx?page=3&TopicRelationlD=1230&ContentlD=125739) 


Ohio held a one-day, statewide Educator Evaluation Symposium and used the event to create online resources 
for its districts. 


U If your state has early-adopter 

or pilot districts testing your 
t new system, ask these district, 

school, and teacher leaders to support 
implementation training for the remaining 
districts in your state. Recorded video 
interviews and presentations from these 
local experts is an efficient way to share 
key, credible insights across a state. 


State 

^Spotlight 


Colorado held multiple in-person events to provide information and training to district leaders. 

■ Eight Educator Effectiveness Symposia (four in June 2011 and four in June 2012) across the state introduced 
district leaders to the state’s legislative and policy changes on educator evaluation. 


■ A one-day Educator Effectiveness Summit (July 2012) convened more than 500 educators from 94 districts to 
learn about the state’s new evaluation policy and implementation timelines; distribute a wide variety of state 
resources for implementing the new reforms at the district level; and hear presentations by and collaborate with 
district teams, experts, and pilot districts. 


Source: Lessons Learned on Communications and Engagement: Colorado Case Study (http://www.tqsource.org/ 
publications/TQ__Policy-to-PracticeBrief_CO_Case_Study.pdf) 
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EDUCATOR ORIENTATION TRAINING 



In the rush to support district leadership teams and 
train evaluators, it can be easy to overlook training for 
the teachers and principals who are being evaluated. 

Informing and preparing educators with detailed 
processes, materials, and tools helps reassure 
concerned educators and dispels myths about the 
new process. States also might consider reaching 
out to regional agencies, professional associations, 
and nonprofit organizations to provide this training. 

States use a variety of methods for educator 
orientation training, such as the following: 

■ Train-the-trainer series. New York used Network 
Team Institutes, which were monthly turnkey 
sessions with regionally based teams, to 
prepare a cadre of more than 800 individuals 
who can train educators on the new evaluation 

systems. Ail resources from the institutes are available online for individual educators 
or teams of educators to access on demand. 


Be sure to include opportunities 
for educators to develop an 
in-depth understanding of 
the leadership or instructional evaluation 
frameworks in your district or state. This ensures 
that district and school teams have a shared 
understanding of the rubric and a common 
vocabulary for discussing evaluation results. 

Rhode Island, for example, provides 
teachers with the Framework forTeaching 
Effectiveness Series, a series of self-paced 
modules for teachers to develop an 
understanding of the state’s Professional 
Practice rubric. 


■ Online, self-paced webinars. Indiana created an online video series and webinars for 
teachers that provide a quick and easy way to learn about the state’s overall system, 
classroom observations, student growth data scores, student learning objectives (SLOs), 
and summative scoring. 


State 

^Spotlight 


value-added modeling and how it is used in the evaluation system, Tennessee created these user-friendly resources to 
translate complex concepts into laymen’s terms to increase understanding and decrease anxiety about the use of this 
student growth measure. 

Sources: The Power of Using Value-Added Analysis to Improve Student Learning: A Guide for Educators (http://team-tn. 
org/assets/educator-resources/Power_of_Using_VA.pdf) and Teacher Model (http://team-tn.org/teacher-model) 


Tennessee provides a variety of resources-from printed guides to online videos-that acquaint educators with the 
Tennessee Value-Added Assessment System. Recognizing the widespread lack of understanding among teachers about 
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EVALUATOR TRAINING 


To cultivate and maintain a pool of skilled and reliable 
evaluators, your state will need a targeted investment 
of time and resources. This investment is critical to 



Consider including a wider 
range of staff in addition 
to superintendents and 


maximize implementation fidelity and ensure that your 


principals in this training, such as district 
administrators, teacher leaders, assistant 
principals, and department heads. This 
creates a diverse and broad pool of 


evaluation system produces accurate, consistent, and 
legally defensible results and ultimately can improve 
teaching practice. Evaluator training extends beyond 
initial training and certification and should include 
ongoing evaluator monitoring with opportunities for 
coaching, recalibration, and strengthening of skills 
across time. 


burden on superintendents and principals 
while increasing teacher buy-in and trust. 


evaluators that can reduce the time 


States generally use a variety of methods, sometimes 
in combination, to provide evaluator training, including 
the following: 

■ In-person training. State education agency, regional, or district staff provide training directly 
to evaluators but usually with facilitation materials, resources, and exercises created by the 
state (often with support from an external vendor). In the state of Washington, the Office of 
Superintendent of Public Instruction trained a corps of Instructional Criteria and Framework 
Feedback Specialists (ICFFSs) to support principals statewide in evaluating teachers. 
Following principals’ training, these ICFFSs will provide ongoing coaching and training and 
will engage in ongoing training themselves to enhance their own understanding — and that 
of evaluators statewide — of both the observable and nonobservable parts of the instructional 
and leadership frameworks that are the basis of educators’ evaluations. 

■ Online training. Evaluators access and complete all training modules, exercises, and any 
certification assessments through an online and secure platform. Examples of this type 
of training are provided in the following state spotlights. 


Illinois contracted with a vendor to produce 10 online training modules (five for teacher evaluators and five for 
principal evaluators) that provide initial training, certification training, and ongoing calibration opportunities. All 
evaluators must complete the 60 hours of online training on their own and are required to pass a test at the 
conclusion of each module. For participants who fail to pass a test after two attempts, both online and in-person 
remediation training is available. 

Source: Growth Through Learning (http://growththroughlearningillinois.org/Home.aspx) 
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Iowa requires that evaluators (for both administrators and teachers) complete iEvaluate, an online, self-paced course 
(_ that participants must complete within a six-week time frame. iEvaluate provides both initial and certification training. 

Because Iowa does not have a statewide model system, the course focuses on the skills any evaluator would need 
regardless of the system design. Participants must complete a series of assignments demonstrating mastery of the 
course content. For administrators, this course counts toward their licensure renewal. For evaluators who have completed 
the basic training, the option of additional training in Fierce Conversations is available, which will ensure that evaluations 
achieve their intent of promoting meaningful conversations that change practice. 

Source: IEvaluate (http://www.educateiowa. gov/index. php?option=com_content&view=article&id=2618&ltemid=5145) 


State 

^Spotlight 


Ohio provides initial, certification, and ongoing calibration training. The state uses regionally based, three-day trainings 
that combine face-to-face sessions and an online assessment that participants must pass to receive an evaluator 
credential. After completing the assessment, the state provides evaluators with continuing access to additional videos 
for ongoing calibration and practice through an online portal. In addition, Ohio contracted with a vendor to provide 
SLO training using a train-the-trainer approach. 


Source: Ohio Department of Education District Educator Evaluation Trainings (http://www.ode.state.oh.us/GD/ 
Templates/Pages/0DE/ODEDetail.aspx?page=3&TopicRelationlD=1230&ContentlD=125741) 


Bonus 

Resource 


HIGH-QUALITY EVALUATOR TRAINING CHECKLIST 


What are the characteristics of high-quality evaluator training? Use the following checklist to guide or evaluate your 
own evaluator training plans at the state level or provide this resource to districts as they design their own evaluation 
training plans. 


Comprehensive 

Does the training cover all aspects of the evaluation process, not just observation? Comprehensive training prepares 
evaluators to do the following: 

□ Conduct pre- and postobservation conferences. 

□ Coach educators and provide feedback for educators at varied levels of performance. 

□ Analyze nonobservation evidence (e.g., artifact review or student or staff surveys). 

□ Understand and analyze student growth data and measures. 

□ Combine multiple measures through a summative scoring process. 

□ Guide the creation of professional development plans. 

□ Manage time and technology to efficiently complete the process. 
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In-depth 

Does the training cover the core knowledge and skills that evaluators need, including the following? 

□ Common sources of bias and strategies for minimizing subjectivity in the evaluation process 

□ The educational philosophy and research base used to develop the instructional or leadership framework 
and observation rubrics 

□ The purpose and the logic for each performance level and scale in the framework or the rubric 

□ The framework or rubric structure and the core performance behaviors included in each dimension or component 

Concrete 

Does the training provide concrete, focused examples of evidence illustrating each point in the framework or the 

rubric scale? For observer training, evaluators should have access to the following: 

□ An extensive library of norm-referenced videos prescored by master evaluators, tiered by level of performance 
(e.g., exemplary and proficient), and covering numerous subject areas and grades 

□ Short, focused video clips to demonstrate a practice or a skill, explain the scoring rationale at different 
performance levels, and practice scoring each subcomponent 

□ Full-length observation videos to practice scoring against the full framework or rubric 

Hands-on 

Does the training provide ample time and opportunity for evaluators to repeatedly practice scoring with immediate 

feedback on the results, such as the following? 

□ In frame-of-reference exercises, participants view an example of evidence, score the example, and discuss their 
reasoning for assigning the score. Trainers correct mistakes or misconceptions, offer advice, and explain the 
expert scorer’s decision. 

□ Targeted training sets focused on hard-to-score sections of the framework or the rubric that evaluators 
commonly struggle to understand and apply. 

Assessed 

Does the training culminate in a rigorous certification test that assesses evaluators’ knowledge, skill, accuracy, 

and reliability in scoring? A rigorous assessment could include the following: 

□ Scoring a full-length observation video 

□ Scoring short video clips and answering multiple-choice questions or other types of test items to assess 
an evaluator’s knowledge of the rubric and ability to match evidence while relying on observation notes 

□ Meeting minimum requirements for rater reliability in scoring 
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Ongoing 


Does the training plan focus on continuous training, assessment, and evaluator practice as much or more than it does 
on initial, upfront training? For example, does it include the following? 

□ Ongoing monitoring of evaluators 

□ On-site coaching for evaluators 

□ Periodic reassessment and recalibration opportunities 


Sources: 

Graham, M. (2011). Measurement: What is inter-rater agreement and how can designers of teacher evaluation systems 
maximize it? (Research synthesis). Rockville, MD: Center for Educator Compensation Reform, Westat. Retrieved 
from http://www.cecr.ed.gov/researchSyntheses/34008_CECR_RS_lnter_Rater_measurement_508.pdf 

McClellan, C., Atkinson, M., & Danielson, C. (2012, March). Teacher evaluator training & certification: Lessons 
learned from the Measures of Effective Teaching project (Practitioner Series for Teacher Evaluation). San Francisco: 
Teachscape. Retrieved from http://www.teachscape.com/binaries/content/assets/teachscape-marketing-website/ 
resources/march_13whitepaperteacherevaluatortraining.pdf 

National Center for Teacher Effectiveness. (2011, October). Supporting rater accuracy and consistency in classroom 
observation [Webinar], Retrieved from http://www.gse.harvard.edu/ncte/news/NCTE_Webinars.php 

Reform Support Network, (n.d.). Race to the Top brief: State development and use of video libraries to improve teacher 
effectiveness. Washington, DC: Author. Retrieved from http://www2.ed.gov/about/inits/ed/implementation-support- 
unit/tech-assist/video-libraries-teacher-effectiveness.pdf 


^ ^ I WANT TO KNOW MORE! 


For more examples or information on this topic, including additional examples of trainings specific to student 
growth measures, please e-mail gtlcenter@air.org. 

Jenni Fetters is a research associate at American Institutes for Research and provides technical assistance 
for the Center on Great Teachers and Leaders. 
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